Data driven testing is a powerful technique for thorough testing of data dependent software systems. The basic idea in data driven testing is to fix a use case, or an instance of the use of software, such as a likely consumer use of software, and run the use case in the context of several data records. When a large number of use cases are tested against a large number of data records, it is convenient to automate the testing process. The automation software is referred to as a test harness.
FIG. 1a illustrates prior art test harness software 101 for data driven testing. The harness 101 can comprise a plurality of test methods 101a-101c, which contain automation for testing particular use cases of software 105. For example, test method 101a might direct software under test 105 to open two data records and merge the contents of the data records into a new record.
The harness 101 can feed data records 100a-100f to the various test methods 101a-101c such that each method test its aspect of the software under test 105 with a variety of data records. Thus, test method 101a can, for example, instruct software under test 105 to merge 100a with another data record into a new record. Test method can then instruct software under test 105 to merge 100b with another data record into a new record, and so on. Similarly, test method 101b can, for example, instruct software under test 105 to process data record 100a according to a specified function, then process data record 100b according to a specified function, and so on for each data record.
Test harness 101 may also compare the actual output 102 from tested software 105 with desired output 103. The desired output 103 may be logged in a specification for the software under test. For example, a very simple software under test 105 may add three units to any data record that is entered. Thus, if the number five is entered, e.g. as a first tested data record 100a, then software under test 105 should output the number eight. The number eight could be stored, e.g., as 103a. in a specification with the desired output 103 for the software under test. The test harness 101 can provide the function of comparing any actual output 102a with the desired output 103a. The test harness may include a process for making this comparison and generating results 104 to indicate which tests were passed, which were failed, and other useful information about the tests that were conducted.
Note that it is often desirable to control the subset of tests that are performed. It is often unnecessary to test software 105 against each and every test method 101a-101c in combination each and every data record 100a-100f. Such a practice may consume too many testing resources: both computing resources in processing all the tests, and human resources in filtering through and analyzing the relevant test results. FIG. 1a is simplified from a more typical situation that may include thousands of data records and thousands of test method variations. To control the number and types of tests that are performed, the data records 100a-100f may be associated with a priority. For some tests, it may be necessary to test only high-priority records, and it is desirable to automatically filter such high-priority records and run only the associated high-priority tests.
Similarly, the test methods 101a-101c may be associated with a priority. For some test runs, it may be considered necessary to test only certain high-priority test methods or test method variations. It is desirable to automatically filter such high-priority test methods and run only the associated high-priority tests.
There are several prior art patterns that a test harness can presently adopt to implement data driven tests. These various patterns each have shortcomings in controlling the particular data records that are used in a test, and in controlling which test method variations are used in a test. Thus, the following prior art test harness patterns do not allow for the level of test control that can be desired in choosing test methods and data records to run in a test.
FIG. 1b presents a set of components that can replace the section enclosed by a dotted line 150 in FIG. 1a. FIG. 1b represents one common prior art test harness configuration: the loop pattern. When a test harness 101 is configured as a loop, data 100 is fed serially to a test method 101a by a loop process 124. The below pseudo-code provides an common exemplary implementation of a loop process 124 for a test harness conducting data driven tests:
for each (DataRecord record in DataRecords){ // Execute Test code for the Data Record}
The pseudo test code sample set forth above is satisfactory when the goal is simply to exercise a particular piece of developer code through all possible sets of test data. Since most software systems have different facets that are tested in the context of every data record, the loop pattern is a sensible solution in many settings.
Though a reasonable first solution to testing a use case against a large amount of data, the loop pattern approach has some serious shortcomings that may arise. These shortcomings, in general, relate to the lack of control over which tests may be desirable to run. More specifically, test granularity and test prioritization are difficult to achieve.
Granularity of implemented tests: The loop pattern is not granular, quantifiable and isolatable. It is therefore useless in generating test reports that are used by management to determine if the code under test is fit for a release. It is also useless for selectively running some tests to determine particular areas for software improvement, such as build quality. The loop pattern approach may thus not satisfactorily meet the needs of professional testing organizations due to its lack of granularity.
Prioritization of tests: The loop pattern does not support any meaningful test prioritization. To generally describe prioritization, the unit of testing in most test organizations is the “test method.” In most test organizations, priorities are associated with test methods to determine the gravity of a particular failure. If a failure of a test method is a “critical failure,” for example, the release of the tested software may be prevented or delayed. Test method priorities could also be used to determine a subset of tests that should pass for build verifications, or subsets that should pass to meet some other requirement.
To accommodate the use of test methods, most test harnesses have the notion of a “test class” which contains test methods. Test classes may fit the standard notion of a class in computer science, such as a Common Language Runtime (CLR) class. Test classes may be given a special attribute which indicates to the test harness that they are test classes. Test methods may be regular member methods of a test class, or test classes, with a special attribute to indicate to a test harness that the test method is a unit of a particular test class. The attribute associated with a test method may also contain a priority for the method.
Further, a test assembly is generally understood as a collection of test classes, along with a special class, often called “test suite” which extends a test harness defined class for the purpose of ease of identification in an assembly. This test suite class can act as container for an instantiated subset of test classes in a test assembly. The test suite may thus inspect its assembly to instantiate all classes marked as a particular test class. The test harness can look for all test suite classes in an assembly, and query each of them to get the test classes contained in the test suite. The test harness can invoke all the test methods of the obtained test classes. Also the harness could collect the results of the test method invocations and generate a report of the test automation.
FIG. 1c illustrates an expanded view of prior art FIG. 1b, and demonstrates that control process 130 cannot access or respond to testing priorities, whether they are data priorities 131 or test method priorities 132, when the harness 101 is configured as a loop.
FIG. 1d presents a set of components that can replace the section enclosed by a dotted line 150 in FIG. 1a. FIG. 1d represents another common prior art practice: the class extension pattern. When a test harness 101 is configured in a class extension pattern, a generic class 125 can take in data records 100a-100f, and implement an appropriate test method 101d-101f. 
The below provided pseudo-code is a common implementation of a class extension pattern generic class 125 for test harnesses that implement data-driven tests. Note that in this pattern, a tester implements a generic class without a special attribute to mark it is a test class. The generic class can takes in a data record in its constructor, and implements test methods, with special attributes that mark them to be a test method, in that class. For every unique data record, the tester can author a test class that simply extends the special class.
// Generic Test Classpublic class GenericImplementation{ public GenericImplementation(DataRecord record){...}; [TestMethod(Id=1,Pri=0)] public ReturnCode Test1( ){...}  [TestMethod(Id=2,Pri=1)] public ReturnCode Test2( ){...}}[TestClass]public class DataRecordTest1:GenericImplementation{ public DataRecordTest1:GenericImplementation(record1){..}}[TestClass]public class DataRecordTest2:GenericImplementation{ public DataRecordTest2:GenericImplementation(record2){..}}....
This test pattern is, in general, better than the loop based pattern. The tests are now granular and quantifiable. Though effective, this pattern has a serious drawback in that any given test method in all the test classes have the same priority. This is because the test methods are pulled into the class via inheritance, and the data record used to test has no input to the priority of a test method.
FIG. 1e illustrates an expanded view of prior art FIG. 1d, and demonstrates that, while control process 133 can control testing based on data priorities 131, the generic implementation of test methods operates to strip test method priorities 132 thereby blocking the use of test method priorities 132 as a means of controlling which tests to run. A test harness that uses the class extension pattern therefore does not allow for effective control over both the data records and the test methods that will be used in a given test run.
Finally, data driven tests may be implemented, using presently available techniques, using the cut and paste pattern. This pattern is not illustrated. Using the cut an paste pattern, a tester implements a test class for a particular data record, and then creates test classes for every data record used in a test. The tester simply cuts and pastes the code from an implemented test class, and modifies the data record and method priorities manually in all the test classes. This pattern enables the tests to be granular and prioritized but it is inelegant, and cumbersome to implement and maintain. A small change in the semantics of one of the test methods will require changes in the corresponding test method in all implemented test classes. When 5000 data records are commonly involved in a test, the work needed to manually alter the test methods using the cut and paste pattern is unacceptable.
There is a need in the art to overcome the aforementioned limitations of current systems and methods for data driven software testing. There is a need to implement more granular, prioritized data driven tests. Moreover, there is a need to do so in such a way as to help testers implement data driven tests effectively and in an easily maintainable fashion with minimal code.